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The  objective  of  this  analysis  is  to  reduce  the  policy  and  management 
process  costs  of  Congressionally  mandated  Live  Fire  Test  and  Evaluation 
procedures  in  the  new  Performance  Based  Defense  Acquisition  environment. 
The  service  and  acquisition  program  independent  Live  Fire  Test  and  Evaluation 
Community  has  not  fully  recognized  the  changing  defense  acquisition 
business  environment  and  modified  policy  and  business  processes  to  account 
for  the  changes.  It  is  not  the  purpose  of  this  paper  to  dispute  the  need  for  live 
fire  test  and  evaluation,  which  is  and  will  remain,  a  key  component  of  the 
defense  acquisition  process  in  order  to  ensure  combat  system  survivability 
and/or  lethality.  This  paper  explores  the  historical  background,  features  and 
resourcing  of  DOD’s  Life  Fire  Test  and  Evaluation  Program.  Further,  the  paper 
examines  how  simulation  and  risk  management  might  be  integrated  to  ensure 
user  confidence  in  weapon  systems,  while  seeking  a  more  effective  and 
affordable  process.  The  paper  makes  four  recommendations  to  that  end. 
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I.  Introduction: 

The  fall  of  the  Berlin  Wall  in  1989  and  the  subsequent  dissolution  of 
most  of  the  world’s  Communist  states  over  the  following  few  years  was  a 
watershed  event  for  the  US  Department  of  Defense  and  all  professionals 
involved  in  the  theorizing,  planning,  and  operations  of  national  defense.  In  a 
very  short  time  span,  defense  professional  thought  has  had  to  transfer  from 
planning  and  operating  in  a  relative  balance  of  power  stasis  to  one  of  relative 
random  events  and  chaos.  There  have  been  many  emerging  results  of  this 
change  in  virtually  every  aspect  of  national  defense. 

Notably,  defense  intellectuals  have  theorized  about  a  broad  array  of 
national  defense  changes  categorized  as  the  Revolution  in  Military  Affairs.  For 
the  purposes  of  this  paper,  the  definition  of  the  Revolution  of  Military  Affairs  will 
be  simply  the  changes  in  military  affairs  brought  on  by  the  advent  of  the 
Information  Age.  These  changes  are  characterized  by  features  such  as  short  to 
no  notice  joint/combined  maneuver  warfare  of  smaller,  more  rapidly 
maneuverable  and  deployable  forces.  These  forces  possess  near  real  time 
situational  awareness  of  friendly  forces,  enemy  forces,  terrain,  and  weather 
and  are  staffed  with  smaller  numbers  of  personnel  equipped  with  high 
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technology  equipment  for  intelligence,  command,  control,  communications  and 
logistics.1  These  forces  are  armed  with  precision  or  “brilliant”  weapons  for 
engaging  the  enemy.2 

The  initial  transition  from  industrial  age  warfare  to  information  age 
warfare  began  during  the  Viet  Nam  War  and  accelerated  through  the  Persian 
Gulf  War  and  is  still  accelerating.3  To  complete  the  transition,  forces  will  have 
to  be  re-equipped  with  information  age  technologies  as  today’s  forces  are  still 
largely  equipped  with  industrial  age  equipment.  To  make  the  transition 
envisioned  by  the  Revolution  in  Military  Affairs  affordable,  and  therefore 
possible,  there  will  have  to  be  a  Revolution  in  Business  Affairs.4  This  revolution 
is  largely  under  way,  and  has  been,  since  the  Clinton  administration  came  to 
office  in  1992.  The  Revolution  in  Business  Affairs  is  the  revolution  that  impacts 
the  “Business  of  Defense”  as  opposed  to  the  operations  and  conduct  of  armed 
conflict.  The  Revolution  in  Business  Affairs  includes  several  sub-aspects 
including: 

•  Re-engineering  Defense  Transportation. 

•  Acquisition  Reform. 

•  Re-engineering  Defense  Finance. 

1  The  Honorable  Robert  M.  Walker  and  General  Dennis  J.  Reimer,  “  United  States  Army  Posture  Statement 
FY99”,  February  1998,  pp.  23-58. 

2  General  John  M.  Shalikashvili,  “Joint  Vision  2010”,  pp.  11-26  and  General  Dennis  J.  Reimer,  uArmy 
Vision  2010”,  pp.  10-17. 

3  Alvin  and  Heidi  Toffler,  War  and  Anti  War ;  Making  Sense  of  Today's  Global  Chaos .  pp.  73-99. 

4  The  Honorable  Robert  M.  Walker  and  General  Dennis  J.  Reimer,  p.  59. 
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•  A  Revolution  in  Logistics. 

•  Others. 

The  Revolution  in  Logistics,  or  the  transition  from  mass  quantity  supply 
of  the  battlefield  under  a  “push”  concept  to  “focused”  logistics,  is  characterized 
by  supplying  precise  unit  needs  at  a  specific  time  and  is  enabled  by 
information  age  technologies.  Examples  of  such  technologies  include  Total 
Asset  Visibility  and  In  Transit  Visibility  which  provide  “Just  in  Time”  inventory  to 
the  battlefield.  The  Revolution  in  Logistics  also  features  supply  delivered  by 
premium  transportation,  such  as  Federal  Express  or  United  Parcel  Service  and 
contractor  provided  combat  systems  support,  such  as  the  Army’s  Apache 
Helicopter  Prime  Vendor  Support  contractor  provided  maintenance  concept. 
These  changes  are  impacting  all  aspects  of  logistics  including  all  phases  of 
supply,  maintenance,  transportation,  and  materiel  acquisition. 

For  the  purposes  of  this  paper,  materiel  acquisition  will  be  considered  a 
sub-component  of  logistics  and  will  include  all  aspects  of  concept 
development,  developmental  research  beyond  the  pure  research  level,  test  and 
evaluation  (developmental,  operational,  and  live  fire),  production  development, 
production,  deployment,  operations  and  support,  and  disposal.  Information 
age  technologies  have  been  impacting  all  aspects  of  material  acquisition  for 
decades. 
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The  Revolution  in  Logistics  impact  on  the  materiel  acquisition  process 
started  with  the  invention  of  the  micro  chip  and  sped  up  with  the  advent  of 
improved  computing  power  and  speed  and  improved  software  capabilities 
such  as  computer  aided  design  and  manufacturing  capabilities.  The  speed  of 
the  change  in  materiel  acquisition  is  progressing  faster  still,  with  the 
maturation  of  simulations,  including  live,  virtual  and  constructive  simulations. 
This  process  is  essentially  known  as  Simulation  Based  Acquisition.5  Anything 
short  of  actual  warfare  has  been  said  to  be  simulation,  but  for  the  purposes  of 
our  discussion,  we  will  discuss  Simulation  Based  Acquisition  as  employing 
three  levels  of  simulation. 

•  Live  Simulation  -  including  engagements  among  actual  military  forces  and 
equipment  with  simulated  weapons  effects. 

•  Virtual  simulation  -  comprising  interactions  among  manned  simulators  of 
weapons  platforms,  operating  in  wholly  synthetic  computer  generated 
environments 

•  Constructive  simulation  -  mathematical  models  of  combat  including  duels 
between  weapons  and  ranging  to  wars  among  nations. 

In  the  world  of  materiel  acquisition  business  operations  processes,  the 
Revolution  in  Logistics  has  manifested  itself  in  the  form  of  a  phenomenon 
known  as  Acquisition  Reform.  In  ten  years,  Acquisition  Reform  has 

5  LTC  Michael  V.  R.  Johnson,  Sr.,  USA  et  al.  Simulation  Based  Acquisition,  pp.  1-1  to  l-4._ 
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significantly  changed  almost  every  aspect  of  materiel  acquisition  business 
process  operations  including  contracting,  program  management,  engineering, 
developmental  test,  operational  test,  manufacturing,  property  administration, 
quality  assurance,  pricing,  acquisition  logistics  and  others  not  mentioned  here. 

Acquisition  Reform  is  intellectually  driven  by  the  concept  posed  by  the 
question,  “What  is  the  “value  added”  by  this  business  process?”  In  other 
words,  Acquisition  Reform  is  a  return  on  investment  mentality  that  seeks  at 
least  the  same  amount  of  benefit  returned,  preferably  much  more,  for 
resources  expended.  In  fact,  this  attitude  is  not  just  permeating  materiel 
acquisition  but  all  aspects  of  resourcing  in  the  federal  budget  process  through 
the  auspices  of  the  National  Performance  Review  and,  of  course,  the 
Revolution  in  Business  Affairs  in  the  Department  of  Defense. 

The  intellectual  concept  of  Acquisition  Reform  has  transformed  the 
Defense  Department’s  materiel  acquisition  process  from  a  “Risk  Avoidance” 
based  process  to  a  “Risk  Managed”  based  process  primarily  for  reasons  of 
affordability.  However,  study  after  study  also  made  it  clear  that  under  a  risk 
avoidance  based  system,  risk  still  could  not  be  avoided  in  the  extremely 
complex  defense  acquisition  process.  Thus,  the  resources  expended  in 
avoiding  risk  were  seen  as  not  providing  a  commensurate  return  on 
investment.  Therefore  a  key  idea  became  to  manage  the  risks  so  that  when 
failure  inevitably  occurs,  even  as  it  does  under  risk  avoidance,  the  failure 
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comes  at  an  acceptable  price  and  orders  of  magnitude  greater  benefits  will  be 
obtained  elsewhere  in  the  materiel  acquisition  business  process.  The 
mentality  change  was  expected  to  yield  better  combat  materiel  systems,  faster 
and  at  less  expense  to  the  national  treasure. 

Usually,  “What  is  the  “value  added”?”,  is  a  very  difficult  question  to 
answer  involving  very  difficult  relationships  to  demonstrate,  if  they  can  even  be 
demonstrated.  Many  attempts  are  being  made  across  the  Department  of 
Defense  such  as;  the  Defense  Capital  Working  Fund,  Activity  Based  Costing, 
Unit  Cost  Management  and  others  to  explain  the  relationship  between 
resources  expended  to  benefit  obtained.  These  efforts  will  continue  as  long  as 
there  is  competitive  tension  for  resources  within  the  federal  government  at 
large  and  within  the  Department  of  Defense  in  particular. 

The  fight  is  on,  and  has  been  for  some  time,  for  every  organization  to 
“get  competitive  or  die”  in  this  environment.  This  environment  was  officially 
declared  by  Vice  President  Gore’s  Reinvention  of  Government  Initiative  which 
grew  out  of  the  National  Performance  Review.  These  efforts  seek  a 
government  that  works  better  and  costs  less.  The  reinvention  efforts  have  been 
characterized  by  a  Performance  Based  Business  Environment.  In  this 
Performance  Based  Business  Environment,  all  organizations  are  endeavoring 
to  show  the  precise  output  of  benefit  for  the  resources  expended.  These 
relationships  are  usually  tracked  through  performance  metrics  for  key 
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business  function  outputs  and  internal  business  operations  in  the  pursuit  of 
effectiveness  and  efficiency.6  The  implementing  concepts  may  be  more  or  less 
based  on  the  Macolm  Baldridge  Quality  Assurance  criteria,  sometimes  tailored 
and  renamed  to  better  align  with  specific  mission  organizations.  Legislators, 
the  Department  of  Defense  and  all  service  material  acquisition  organizations 
have  been  major  players  in  these  pursuits  with  one  notable  exception  -  the  Live 
Fire  Test  &  Evaluation  Community. 

The  balance  of  this  paper  will  attempt  to  explore  why  the  live  fire  test  & 
evaluation  business  process  has  largely  been  a  non-participant  in  the 
Revolution  in  Business  Affairs  and  Acquisition  Reform  and  why  the  community 
may  feel  exempt.  The  paper  will  offer  some  theories  concerning  why  the  Live 
Fire  Test  and  Evaluation  Community  should  participate  in  Acquisition  Reform 
and  how  that  can  occur.  Finally  some  proposals  will  be  given  to  stimulate 
future  thought  and  action  on  the  matter.  To  get  to  that  point  the  paper  will 
discuss  the  legislative  and  policy  background  of  why  we  have  live  fire  test  and 
evaluation,  how  the  live  fire  test  and  evaluation  business  process  works,  and 
how  the  new  era  in  materiel  acquisition  is  impacting  that  process,  perhaps 
without  even  the  knowledge  of  or  understanding  the  impact  by  the  players,  and 
how  the  Live  Fire  Test  and  Evaluation  Community  can  leverage  the  changes  to 
remain  relevant  and  affordable  in  the  future. 


6  LTC  Dennis  Drayer,  USAF,  “PBBE  -  A  Business  Vision  We  Can  Live  With  ”,  pp.  57-67. 
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This  paper  and  recommendations  may  not  be  well  received.  Criticism  is 
likely  from  the  Live  Fire  Test  and  Evaluation  Community  because  it  suggests 
what  may  be  perceived  as  change  that  threatens  their  business.  Nothing  could 
be  further  from  the  truth.  Therefore,  the  reason  for  undertaking  this 
investigation  is  in  order  to  explain  the  paper’s  position  in  advance. 

First,  the  author  is  not  a  player  in  the  live  fire  test  and  evaluation  process, 
but  is  an  acquisition  professional  and  interested  in  live  fire  test  and  evaluation 
process.  The  concept  of  destroying  something  to  make  it  better  is  inherently 
fascinating.  The  process  has  been  effective  in  the  past.  The  objective  is  to 
keep  it  affordable,  effective  and  relevant  in  the  future. 

On  a  superficial  level  live  fire  test  and  evaluation  is  a  fundamentally  good 
idea  and  has  generally  worked  in  the  past  -  but  the  environment,  particularly  the 
resourcing  environment,  is  rapidly  changing.  Still,  almost  any  process  can  be 
made  better.  This  paper  will  demonstrate  a  strong  proponency  of  live  fire  test 
and  evaluation,  but  in  a  different  form  . 

Second,  the  author  has  been  a  benefactor  of  the  live  fire  test  and 
evaluation  process  as  a  career  combat  arms  military  officer.  This  paper  takes 
the  position  that  there  is  nothing  more  important  than  the  welfare  and 
survivability  of  our  military  personnel.  To  quote  Sun  Tzu, 
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“For  to  win  one  hundred  victories  in  one  hundred  battles  is  not  the  acme  of 
skill.  To  subdue  the  enemy  without  fighting  is  the  acme  of  skill.”.7 

Short  of  that,  an  operation  that  does  not  incur  any  casualties  is  ideal.  We 
have  come  close  to  that  in  recent  years,  especially  in  the  Persian  Gulf  War  and 
in  our  operations  in  the  Balkans.  We  can’t  always  expect  this  to  be  the  case. 

This  paper  is  an  unbiased  look  at  the  live  fire  test  and  evaluation 
process  because  the  author  of  this  paper  is  a  fundamentally  interested  casual 
observer  with  no  ax  to  grind.  The  author  is  not  and  never  has  been  a  program 
manager.  Nor  is  he  likely  to  be  a  program  manager.  In  that  respect,  the  author 
can’t  be  accused  of  bias  against  the  live  fire  test  process  by  viewing  it  as 
adversarial.  Further,  the  author  is  not  a  test  and  evaluation  specialist.  Nor  is 
he  ever  to  likely  be  one.  So  in  that  respect,  he  can’t  be  accused  of  having  a 
bias  for  the  current  live  fire  test  process,  worried  about  threats  to  personal 
vested  interests. 

As  a  soldier  and  former  leader  of  some  of  our  nation’s  best  combat 
soldiers,  the  author  only  wants  our  combat  systems  to  have  maximum  lethality, 
survivability,  operational  effectiveness  and  suitability.  Live  fire  test  and 
evaluation  is  a  critical  component  of  ensuring  those  qualities  in  the  equipment 
we  develop  and  procure.  Therefore,  this  paper  strongly  supports  the  live  fire 
test  and  evaluation  concept  and  argues  that  it  is  a  critical  materiel  business 

7  Sun  Tzu,  The  Art  of  War,  p.  77. 
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process  that  can  not  be  eliminated  under  any  circumstances.  That  is  not  the 
same  thing  as  saying  that  the  process  is  not  ripe  for  reform  or  change. 

As  an  acquisition  professional,  the  author  seeks  the  combat  readiness 
of  our  armed  forces  through  the  acquisition  of  the  most  capable  and  affordable 
equipment  for  the  fewest  possible  dollars,  or  best  value,  so  that  those  dollars 
may  be  better  spent  elsewhere  within  the  Department  of  Defense  and  on  the 
needs  of  the  nation  outside  of  the  Department. 

Therefore  the  changes  that  are  affecting  every  other  materiel  acquisition 
process  under  Acquisition  Reform  must  also  affect  the  live  fire  test  and 
evaluation  process  for  it  to  remain  relevant,  effective  and  affordable.  This  paper 
will  argue  that  the  process  is  not  currently  relevant,  effective  and  affordable. 
Worse  yet,  the  author  could  not  even  identify  any  critical  thinking  going  on  about 
how  to  ensure  relevancy,  effectiveness  and  affordability  happen  in  the  live  fire 
test  and  evaluation  process,  in  spite  of  extensive  research. 

The  appearance  then,  if  not  the  reality,  is  that  the  Live  Fire  Test  and 
Evaluation  Community  envision  themselves  as  unique  and  therefore  exempt 
from  Acquisition  Reform.  Many  communities  have  viewed  themselves  as 
“special”  and  unique  and  therefore  exempt  from  change,  yet  many  that  had  that 
view  have  since  reformed. 
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In  summation,  this  paper  is  written  as  an  “outside  the  box”  think  piece 
intended  to  stimulate  discussion  among  thinking  professionals  concerning  a 
very  interesting  and  critical  topic  and  to  motivate  constructive  change  for  the 
benefit  of  our  nation’s  warriors. 

II.  Background  and  History  of  Live  Fire  Test  and  Evaluation: 

Live  Fire  Test  and  Evaluation  is  defined  as  a  sub  category  of  Test  and 
Evaluation,  specifically  Operational  Test  and  Evaluation,  that  tests  and 
evaluates  the  realistic  system  survivability  (or  in  the  case  of  missiles  and 
munitions  the  system  lethality)  of  Department  of  Defense  acquisition 
programs.  This  is  usually  accomplished  by  taking  live  shots  at  a  system  or 
with  by  system  at  realistic  targets  under  expected  combat  conditions. 

The  proponents  of  the  Department  of  Defense  Live  Fire  Test  and 
Evaluation  Program  market  themselves  as  the  conscience  of  the  acquisition 
system.8  This  is  a  worthy  ideal.  Why  do  we  have  live  fire  test  and  evaluation? 
We  have  it  for  good  and  valid  reasons.  The  history  of  our  nation’s  armed  forces 
is  replete  with  examples  of  weapon  systems  that  did  not  work  in  combat  as 
designed  or  expected,  to  the  detriment  of  our  warriors.  In  World  War  II 
torpedoes  failed  to  explode,  jeopardizing  submariners  engaged  in  mortal 
combat  with  Japanese  surface  warfare  units.  Early  in  World  War  II,  US  tanks 


8  James  F.  O’Bryon,  “Overview  of  Live  Fire  Test  and  Evaluation",  Fort  Belvoir,  VA,  undated,  slide  000035. 
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were  powered  by  gasoline,  which  would  ignite  into  an  inferno  when  hit  by 
enemy  munitions.  Early  in  the  Korean  War,  US  anti-tank  rocket  munitions 
would  bounce  off  of  Soviet-made  T-34  Tanks.  In  Viet  Nam,  US  and  Army  of  the 
Republic  of  Viet  Nam  infantry  would  ride  on  top  of  Ml  13  personnel  carriers, 
outside  of  the  vehicle  and  exposed  to  small  arms  fire,  for  fear  that  exploding 
mines  would  turn  the  inside  of  the  carriers  into  death  traps,  thus  rendering  the 
inside  of  the  carrier  useless  for  protection,  its  primary  purpose.  How  does  this 
happen? 

Realistically,  designers  can  never  be  sure  systems  will  perform  as 
expected  without  extensive  testing,  including  live  fire  testing,  followed  by 
prolonged  operational  use,  including  combat.  Further,  the  laws  of  physics 
apply  and  it  is  almost  impossible  and  unrealistic  to  optimize  engineering  of 
combat  system  design  for  every  threat.  Designs  have  to  optimized  for  the 
“most  likely”  expected  threats.  The  rest  of  the  threats  must  be  addressed 
through  doctrine,  training  and  tactics.  Moreover,  it  is  impossible  to  predict  all 
outcomes  of  combat  engagement  due  to  the  synergistic  effects  of  engagement 
variables  based  on  any  one  engagement  condition  set,  including  crew  factors, 
daylight,  weather,  range,  weapon  system  cant  and  unlimited  other  factors. 
However,  survivability  and  lethality  risk  can  be  improved  through  design,  test, 
production,  doctrine,  tactics  and  training  for  deployment. 
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The  desire  to  reduce  survivability  risk  and  increase  lethality  is  how  live 
fire  test  requirements  came  to  be.  We  do  gain  first  order  insights  from  the 
results  of  live  fire  tests.  Theoretically,  live  fire  test  and  evaluation  is  a 
fundamentally  sound  concept.  But  are  the  current  process  results  adequate  for 
the  expenditure  level,  in  terms  of  manpower,  time,  facilities,  equipment  and 
dollars?  Is  live  fire  test  and  evaluation  currently  a  good  investment?  Can  live 
fire  test  and  evaluation  accomplish  the  stated  objectives  of  ensuring  that  battle 
damage  tolerance  and  damage  control  of  our  crew-carrying  combat  systems  to 
actual  threat  weapons  is  known  and  acceptable  and  that  the  lethality  of  our 
conventional  weapons  (non-nuclear,  biological,  or  chemical)  against  actual 
threat  systems  is  known  and  acceptable?9 

A  parallel  business  process  to  Live  Fire  Test  and  Evaluation  is  practiced 
in  Civilian  Industry.  For  example,  automobiles  are  crash  tested  under  the 
supervision  of  the  National  Highway  Traffic  Safety  Administration  to  determine 
crash  worthiness  and  passenger  survivability.10  The  major  difference  is  that 
automobiles  are  produced  at  much  greater  volume  and  much  lower  costs  than 
the  average  Department  of  Defense  weapon  systems  covered  by  Live  Fire  Test 
and  Evaluation  requirements.  The  average  automobile  costs  in  the  twenty  to 
forty  thousand  dollar  range  per  automobile  and  defense  systems  covered  by 


9  Defense  Systems  Management  College  Fact  Sheet  6.3,  “Live  Fire  Test  and  Evaluation  (LFT&E),  April 
1995,  pp.  6.3-1  to  6.3-2. 

10  James  F.  O’Bryon,  undated,  slide  000132  and  000102. 
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the  live  fire  test  and  evaluation  process  usually  cost  in  the  millions  of  dollars 


per  copy. 

Therefore,  destructive  testing  by  crashing  civilian  automobiles  pays  a 
much  higher  return  on  the  investment  for  destructive  testing  than  it  can  on 
expensive  Department  of  Defense  systems.  The  return  on  investment 
advantage  of  the  automobile  industry  is  intensified  when  considering  the 
widespread  and  constant  usage  of  automobiles  by  billions  of  people  around 
the  world.  In  short,  the  comparison  between  the  two  is  specious  at  best.  The 
fact  that  the  Return  on  Investment  for  live  fire  test  can’t  rival  that  of  the 
automobile  industry  is  still  no  reason  not  to  have  live  fire  testing  of  military 
systems  though,  if  lives  can  be  saved  in  combat  and  training  -  presuming  the 
end  state  of  the  testing  provides  us  with  what,  and  enough  of,  what  we  need  to 
know  to  make  a  difference  and  provide  an  overmatch  of  operational 
advantages  relative  to  future  adversaries. 

The  worthy  objectives  of  live  fire  test  and  evaluation  are: 

•  To  enable  Defense  Department  decision  makers  to  make  informed 
defense  system  acquisition  decisions  at  the  appropriate  time  in  the  system 
life  cycle. 

•  To  gain  potential  insights  in  to  design  flaws  to  provide  opportunity  for 
corrective  action  prior  to  full  rate  production  (albeit  late  in  the  process). 
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•  To  understand  system  survivability  and  lethality  against  expected  threats, 
and  emphasize  individual  soldiers,  sailors,  airmen,  marines  and  crews  of 
each. 

In  short  the  purpose  is  to  save  lives  in  combat.11  I  submit  the  short 
purpose  in  the  previous  sentence  mitigates  the  need  for  any  specific  return  on 
investment.  Still,  we  can  conduct  the  process  much  better  than  we  currently  do. 

Live  fire  test  and  evaluation  is  codified  in  law  under  Title  X  United  States 
Code,  Chapter  139,  Sections  2362,  2366  and  2399. 12  Since  Fiscal  Year  1986, 
when  initially  placed  into  law  in  the  Fiscal  Year  1986  Defense  Authorization  Act, 
the  coverage  of  systems  has  grown  to  cover  more  and  more  defense  systems. 
Coverage  has  expanded  from  Tracked  and  Wheeled  Vehicles  in  Fiscal  Year 
1986  to  All  major  combat  systems  (Acquisition  Category  I  or  II)  in  Fiscal  Year 
1987,  to  the  US  Air  Force  C-17  Air  Transport  in  Fiscal  Year  1993,  to  Theater 
Missile  Defense  in  Fiscal  Year  1994.  Coverage  today  stands  at  eighty  systems 
in  all  services13  The  Department  of  Defense  established  policy  guidance  to 
implement  the  law  in  Department  of  Defense  Directives  5000.1  and  5000.2-M, 
Department  of  Defense  Regulation  5000.2-R,  and  Department  of  Defense 
Instruction  5000.2.14 


11  Ibid.,  slide  000263. 

12  Defense  Systems  Management  College  Fact  Sheet  6.3,  p.  6.3-1. 

13  James  F.  O’Bryon,  slides  000030  and  000044a. 

14  Defense  Systems  Management  College  Fact  Sheet  6.3,  p.6.3-1  and 
http: //www.  dote .  osd.  mil/lfte/BBO  OK.  htm 


15 


Along  with  the  growth  in  coverage  in  Fiscal  Year  1989,  the  Defense 
Authorization  Act  authorized  the  Secretary  of  Defense  to  Reprogram  0.3%  of 
system  procurement  funds  to  fund  Live  Fire  Test  and  Evaluation.15  In  Fiscal 
Year  1994,  the  Federal  Acquisition  Streamlining  Act  transferred  all  Live  Fire 
Test  to  the  responsibility  of  the  Department  of  Defense  away  from  the 
services.16 


All  of  this  legislation  was  enacted  and  had  the  cumulative  effect  of 
significantly  increasing  the  allocation  of  resources  to  live  fire  test  and 
evaluation  activities,  at  the  very  time  that  system  procurement  funds  were 
plummeting  to  around  thirty  billion  dollars  per  year  for  the  entire  DOD,  a 
reduction  in  excess  of  52%  from  1989  levels.17  GEN.  Shalikashvili,  in  his  last 
Chairman’s  Program  Assessment  as  Chairman  of  the  Joint  Chiefs  of  Staff, 
indicated  that  to  maintain  adequate  readiness,  the  level  of  procurement 
needed  to  be  at  least  at  sixty  billion  dollars  per  year.18 


We  are  still  not  at  the  recommended  procurement  level  today  at  about 
forty  eight  billion  per  year  in  the  current  program.  It  is  unlikely,  given  competing 
demands  in  the  federal  budget,  and  even  within  the  Department  of  Defense, 


15  James  F.  O’Bryon,  slide  000030. 

16  Ibid.,  slides  000005  and  000030. 

17  “ America' s  Army . Into  the  21st  Century  ”,  Army  Program  Analysis  and  Evaluation  Directorate,  p.6. 

and  the  March  24,  1999  Statement  on  Modernization  of  the  US  Army  by  General  Dennis  J.  Reimer,  Chief 
of  Staff,  Army,  before  the  Subcommittee  on  Airland  Forces,  Committee  of  the  Armed  Services,  United 
States  Senate,  First  Session,  106th  Congress,  p.  4. 
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that  the  Department  of  Defense  will  reach  the  sixty  billion  dollar  per  year  level 
for  procurement  for  years  to  come,  if  ever.  Therefore,  it  is  most  interesting  that 
live  fire  test  and  evaluation  has  been  a  growth  process  in  an  era  of  steeply 
diminishing  Research  Development  Test  &  Evaluation  and  Procurement 
dollars,  thereby  consuming  an  ever  greater  share  of  an  ever  diminishing 
resource  base! 

Thus,  not  only  was  the  timing  of  the  Live  Fire  Test  legislation  financially 
inopportune,  the  effectiveness  of  the  legislation  is  called  into  question  since 
the  Department  is  developing  very  few  new  weapons  (and  under  different 
acquisition  processes  defined  by  the  Performance  Based  Business 
Environment)  and  because  the  Department  can  not  buy  enough  to  sustain 
operational  readiness  of  current  systems  at  any  rate. 

This  is  not  a  unsolvable  problem  under  current  live  fire  test  and 
evaluation  paradigms  though,  as  it  could  be  fixed  with  additional  funding,  given 
a  turn  around  in  the  resourcing  climate.  Recent  developments  in  the 
Washington  DC  environment  indicate  that  the  resourcing  climate  may  be 
improving,  even  if  incrementally.  The  question  is,  will  the  0.3%  of  procurement 
funds  be  effectively  used  and  accomplish  the  worthy  intent  of  the  Live  Fire  Test 
and  Evaluation  program? 


18  1997  Chairman’s  Program  Assessment. 
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At  $60  billion  of  procurement,  live  fire  test  and  evaluation  funds  should 
be  about  $1 .8  billion  at  the  0.3%  rate  if  the  Secretary  of  Defense  applies  the 
0.3%  reprogramming  to  the  entire  Department  of  Defense  procurement 
allocation.  If  the  Secretary  allows  reprogramming  of  only  the  procurement 
funds  for  the  covered  eighty  systems,  then  the  available  funds  for  live  fire  test 
are  much  less. 

Even  with  the  most  optimistic  funding  projections,  since  most  of  the 
covered  eighty  systems  cost  in  the  millions  of  dollars,  how  much  destructive 
testing  of  full-up  systems  can  be  accomplished  at  approximately  $22.5  million 
per  covered  system?  Obviously  not  much,  unless  accomplished  at  the 
component  level  or  by  some  means  other  than  destroying  full  up  low  rate  initial 
production  systems,  which  is  when  the  first  full  up  systems  are  to  be  destroyed, 
and  therefore  the  maximum  amount  of  live  fire  test  and  evaluation  activity  is  to 
be  expected.  As  of  October,  live  fire  test  and  evaluation  under  the  statutory  and 
policy  guidance  had  been  completed  on  twenty  five  of  eighty  systems.19 

Upon  further  analysis,  we  find  that  live  fire  test  also  includes  Joint  Live 
Fire  Test  and  Evaluation,  after  the  production  decision  and  during  upgrade 
activity  later  in  the  system  life  cycle20  The  difference  in  funding  above  the  0.3% 
of  procurement  funds  is  to  be  paid  from  service  research,  development,  test 


19  James  F.  O’Bryon,  slide  000044a. 

20  Ibid.,  slide  000093. 
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and  evaluation  or  procurement  accounts,  which  are  already  woefully 
inadequate. 

Furthermore,  live  fire  test  and  evaluation,  by  the  community’s  own 
definition,  is  a  process,  not  an  event.21  Activity  is  supposed  to  increase  over 
time  during  operational  testing  to  sequentially  cover  individual  components, 
then  multiple  components,  followed  by  subsystems,  prototypes  and  full  up 
systems  in  sequence.22  This  presumably  would  be  accomplished  on  the 
theoretically  programmed  $22.5  million  for  each  of  the  eighty  systems.  This 
funding  level  and  requirement  presents  a  considerable  management 
challenge  indeed. 

What  we  have  then  at  the  end  of  this  analysis  is  a  process  where  there 
is  a  total  strategy  versus  resources  mismatch  with  resources  being  completely 
inadequate  to  accomplish  the  stated  objectives.  Worse  yet,  there  are  no 
prospects  of  the  stated  objectives  ever  being  adequately  resourced.  Someone 
needs  to  tell  the  emperor  he  has  no  clothes. 

Live  Fire  Test  and  Evaluation  focuses  on  determining  the  acceptability  of 
system  vulnerability  and  crew  safety,  or  the  probability  of  being  killed  if  hit,  also 
known  as  Pk/h.  System  survivability  (Ps)  is  defined  by  the  relationship  1  -  (the 
probability  of  being  attacked  given  an  encounter  (also  known  as  Pa/e)  times  the 

21  Ibid.,  slides  000093,  000047,  000017,  000050. 
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probability  of  being  hit  given  an  attack  (also  known  as  Ph/a)  times  Pk/h,  or 
restated,  Ps  =  1  -  (Pa/e  X  Ph/a)  X  Pk/h.23  This  is  a  complex  outcome  based  on 
the  relationship  of  type  and  power  of  munitions  in  the  attack,  shot  lines,  points 
of  impact,  and  terminal  munitions  effects,  and  is  very  difficult  to  predict  without 
destructive  testing.  The  outcome  is  essentially  valid  only  for  a  single  selected 
set  of  test  criteria  since  changing  munitions,  shotline,  point  of  impact,  or  “fog  of 
war”  would  alter  the  synergistic  terminal  effects  of  the  engagement.  The  same 
set  of  conditions  apply  to  lethality  testing  of  munitions,  only  in  reverse. 

Moreover,  live  fire  test  focuses  only  Pk/h  which  is  a  dependent  variable  of  Pa/e 
and  Ph/a.  So  clearly  Pk/h  can  be  mitigated  by  a  wide  variety  of  factors  such  as, 
tactics,  doctrine,  weather,  daylight  and  many  more  factors  other  than  system 
design.  This  gives  the  live  fire  test  and  evaluation  process  an  exceedingly 
difficult  set  of  factors  under  which  to  prove  (or  disprove)  system  survivability  or 
lethality. 

III.  Acquisition  Reform 

Enter  Acquisition  Reform  and  overlay  the  effects  of  Acquisition  Reform 
upon  the  requirements  and  developments  of  the  live  fire  test  and  evaluation 
legislation  and  program  of  the  Department  of  Defense.  With  the  ascendancy  of 
the  Clinton  administration  in  1992,  everything  in  the  defense  acquisition 
process  was  soon  to  change.  One  of  the  first  acts  of  the  administration  was  to 


22  Ibid.,  slides  000017  and  000050. 


20 


conduct  the  National  Performance  Review  which  led  to  the  Reinvention  of 
Government  Initiative.  In  the  Defense  Department  this  manifested  itself  in  the 
Revolution  in  Business  Affairs  and  Acquisition  Reform,  discussed  earlier. 
Acquisition  Reform  touches  every  single  aspect  of  the  Defense  acquisition 
process  including:  concept  development,  engineering,  developmental  and 
operational  testing,  contracting,  property  administration,  quality  assurance, 
packaging,  transportation,  disposal  and  others  not  mentioned  here.  In  most 
regards  it  has  manifested  itself  by  slashed  budgets  and  large  reductions  in 
acquisition  personnel  to  be  replaced  by  information  age  capabilities,  such  as 
digitizing  business  processes  and  simulations.  Has  acquisition  reform 
touched  the  Live  Fire  Test  and  Evaluation  Program  of  the  Department  of 
Defense?  It  would  appear  that  it  has  not  in  any  substantive  way. 

The  overriding  change  in  acquisition  reform  has  been  fostered  by  the 
advent  of  the  Performance  Based  Business  Environment.  The  Performance 
Based  Business  Environment  creates  a  vision  of  a  quality,  business  -  like 
environment  that  simplifies  and  takes  advantage  of  the  basic  acquisition  and 
sustainment  tools  to  enhance  the  products  we  provide  to  the  warfighter.24  It  is 
characterized  by  the  following: 

•  Dual  use  products  and  processes. 


23  Ibid.,  slide  000084. 

24  LTC  Dennis  Drayer,  p.  58. 
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•  World  class  processes. 

•  Commercial  state  of  the  art  technologies. 

•  Integration  of  commercial  and  military  development. 

The  end  state  objective  is  to  provide  better,  faster,  less  expensive 
systems  and  smoother  operations.25  Philosophically,  the  Performance  Based 
Business  Environment  shifts  the  defense  acquisition  process,  including  test 
and  evaluation,  from  a  process  of  Risk  Avoidance  to  Risk  Management.26  The 
objectives  are: 

•  Convey  product  definition  and  key  process  expectations  (specifications)  to 
industry  in  terms  of  performance. 

•  Promote  life  cycle  systems  engineering  and  management  practices 
(including  integrated  product  development  process  and  support). 

•  Increase  emphasis  on  past  performance. 

•  Shift  from  a  cost  -  based  (plus  profit)  to  a  price  based  procurement  system 
(including  profit). 

•  Motivate  process  efficiency  and  effectiveness  up  and  down  the  entire 
supplier  base. 

•  Encourage  life  cycle  risk  management  instead  of  risk  avoidance. 

•  Simplify  acquisition  and  support  operating  methods.27 


25  Ibid.,  p.58. 

26  Ibid.,  p.60. 

27  Ibid.,  p.60. 
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Nowhere  was  test  and  evaluation  exempted  in  this  philosophical 
change.  The  fundamental  shift  in  the  Performance  Based  Business 
Environment  was  the  advent  of  performance  specifications  in  lieu  of  military 
specifications  and  standards.  What  has  this  done  to  the  defense  acquisition 
community?  First,  it  has  shifted  responsibility  for  configuration  control  and 
technical  data  package  control  from  the  government  to  industry.  Instead  of 
control,  product  managers  and  requiring  agencies  have  only  insight  and 
influence.  No  longer  can  we  tell  the  contractors  how  to  do  the  thing,  we  can 
only  tell  them  the  thing  we  want  to  do. 

This  new  philosophy  has  been  in  effect  in  the  Army  since  1992.  If  the 
contractor  feels  he  has  a  better,  less  expensive,  faster  product,  material  or 
process  then  it  may  be  adopted.  The  program  manager  is  only  to  be  informed 
and  there  is  little  the  program  manager  can  do  unless  the  changes  violate  the 
fundamental  performance  requirements  e.g.  size,  weight,  speed,  altitude,  cost 
or  price  and  others.  In  former  times  making  such  changes  could  even  be 
illegal  and  prosecutable  under  fraud  legislation  for  product  and  process 
material  substitutions. 

Acquisition  Reform  and  the  use  of  performance  specifications  has  also 
collided  with  the  dearth  of  sustainment,  research  and  procurement  funding  to 
create  spin  -  off  concepts  and  programs  such  as  modernization  through 
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spares  and  technology  insertion  or  applique.  In  modernization  through  spares, 
a  combat  system  will  be  modernized  when  major  components  or  sub¬ 
components  are  replaced.  This  is  accomplished  by  replacing  an  older  engine 
with  a  new  more  modernized  engine  in  a  tank  or  helicopter,  for  example.  Other 
types  of  modernization  can  occur  to  fire  control  systems,  protection  or 
survivability  systems,  communications  systems,  and  any  other  subsystem 
below  the  top  level  combat  system. 

In  technology  insertion  or  applique,  a  new  lower  tier  system  that  may  not 
have  been  a  part  of  the  original  top  level  system  is  inserted  into  the  top  level 
combat  system  or  overlaid  on  the  top  level  combat  system,  provided  there  is  an 
adequate  space,  weight  or  size  allowance.  An  example  of  this  would  be  the 
Army’s  ongoing  Horizontal  Technology  Integration  effort  to  digitize  Army  combat 
forces.  The  intent  of  both  of  these  concepts  is  to  make  the  current  existing  top 
level  system  more  capable  in  an  era  when  there  is  inadequate  funding  in  the 
Defense  Budget  to  develop  completely  new  systems  from  a  paper  concept. 

Both  approaches  are  very  pragmatic  and  sound  given  the  current  acquisition 
resourcing  environment. 

But  what  is  the  synergistic  impact  of  acquisition  reform  efforts  on 
Combat  system  configuration,  configuration  control,  and  stability?.  In  short,  the 
impact  is  phenomenal.  The  US  Air  Force  has  recognized  for  years,  even  before 
Acquisition  Reform,  that  in  aircraft  production  lines,  combat  system 


improvements  were  being  applied,  often  to  almost  every  new  aircraft  on  the 
production  line.  This  situation  led  to  a  case  where  almost  no  two  aircraft  were 
the  same.  Acquisition  Reform  has  accelerated  this  process,  and  looks  to 
accelerate  the  effect  even  more  in  the  future,  as  many  new  defense  contractors 
are  coming  on  line  with  performance  specifications  and  are  operating  in  the 
performance  based  business  environment.  The  extended  life  of  combat 
systems  in  the  current  resource  constrained  environment  will  only  amplify  the 
impacts  of  Acquisition  Reform  on  combat  system  configuration  due  to  the 
Performance  Based  Business  Environment,  Technology  Insertion, 
Modernization  Through  Spares  and  other  such  efforts. 

This  rapid  application  and  acceleration  of  the  Performance  Based 
Business  Environment  impact  every  aspect  of  Defense  acquisition: 
contracting,  program  management,  quality  assurance,  production,  acquisition 
logistics,  developmental  and  operational  test.  It  also  impacts  live  fire  test  & 
evaluation  in  a  tremendous  way. 

IV.  Live  Fire  Test  &  Evaluation  and  Acquisition  Reform. 

As  previously  discussed  in  Section  II  of  this  paper,  the  science  and 
mechanics  of  live  fire  test  and  evaluation  is  largely  one  of  terminal  effects  of 
combat  engagement  and  impact  dynamics,  which  is  very  complex.  Acquisition 
Reform  has  made  them  more  complex  than  before.  Results  of  the 
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engagement  and  impact  dynamics  is  highly  synergistic  and  depends  on  multi- 
varibale  analysis  and  is  difficult  to  predict.  It  is  known  however,  that  interactions 
between  sub-components  of  the  system  design  is  what  yield  the  survivability  or 
lethality  results.  Why  then,  do  we  place  so  much  reliance  and  resource  so 
heavily  on  the  result  of  one  specific  solution  set  of  variables  when  that 
particular  solution  set  has  a  low  probability,  approaching  zero,  of  ever  being 
repeated  again?  Not  only  is  the  variable  solution  set  unlikely  to  be  repeated,  it 
is  highly  difficult  to  replicate,  which  is  a  key  requirement  of  science.  However, 
since  testing,  though  not  a  statistical  exercise,  is  based  in  probabilities  and 
distributions  to  certain  scenarios,  i.e.  60%  of  all  engagements  will  be  frontal 
shots,  at  4  to  6  km  range,  traveling  at  15  mph...,  live  fire  testing  should  be  able 
to  baseline  a  few  scenarios  with  a  level  of  confidence  and  then  iterate  using 
simulation  to  examine  the  “what  ifs”  and  the  outliers. 

The  replication  difficulty  comes  from  two  sources  that  I  will  call 
environmental  factors  and  hardware  factors.  First,  I  will  refer  to  the 
environmental  factors  as  the  munitions  used,  the  range,  the  weather,  the  shot 
line,  the  point  of  impact,  crew  actions  and  many  other  such  variables.  Combat 
and  maneuver  are  highly  dynamic.  The  same  set  of  environmental  criteria  will 
likely  never  be  replicated  on  the  battlefield;  even  if  what  we  did  on  the  range 
could  be  replicated  with  a  high  degree  of  consistency,  the  range  would  be 
irrelevant  to  actual  combat. 
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Second,  acquisition  reform  has  injected  a  whole  new  set  of  variables, 
external  to  all  of  the  environmental  factors,  which  I  will  refer  to  as  hardware 
factors.  That  is  to  say  that  no  two  future  combat  systems  are  likely  to  have 
exactly  the  same  configuration  and  Technical  Data  Package.  Even  though  this 
is  true,  all  of  the  substitutions  still  have  to  meet  performance  requirements  e.g. 
form,  fit  and  function.  The  rate  of  configuration  change  between  systems  is  the 
key  figure  of  merit  that  needs  to  be  investigated.  Because  no  two  systems  are 
likely  to  have  the  same  configuration  under  Acquisition  Reform,  the  effect  is  to 
change  the  internal  component  impact  dynamics  between  every  “similar” 
system.  In  short,  no  two  tanks  are  likely  to  be  configured  the  same  in  the  future, 
so  where  is  the  “value  added”  for  a  single  live  fire  test  and  evaluation  event, 
costing  millions  of  dollars,  on  one  specific  system  that  is  different  than  one  that 
rolled  off  of  the  production  line  after  it?  It  is  my  contention  that  the  “value  added” 
due  to  environmental  and  hardware  factors  is,  while  not  insignificant,  severely 
limited.  To  continue  the  current  live  fire  test  and  evaluation  process  into  the 
future  without  significant  improvements  is  to  ignore  reality  and  fail  to  obtain 
maximum  benefit  from  our  scarce  resourcing. 

Proponents  of  the  current  live  fire  test  and  evaluation  process  would 
respond  to  my  contention  that  statistical  significance  for  a  Live  Fire  Test  and 
Evaluation  Event  is  unnecessary.  The  live  fire  test  and  evaluation  proponents 
contend  that  we  learn  much  from  a  single  shot.28  I  agree  with  this  statement 

28  James  F.  O’Bryon,  slide  000290. 
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but  it  ignores  the  issue.  “Much”  is  a  very  subjective  term.  Usually,  in  munitions 
testing  of  Precision  Guided  Munitions,  at  least  50  expensive  shots  are  required 
to  gain  only  a  statistically  insignificant  70%  to  80%  confidence  level.29  The 
issue  is  are  we  obtaining  the  most  “value  added”  from  that  single  shot  (or  50), 
and  the  expenditure  that  goes  with  it?  I  contend  that  we  are  not  and  process 
reengineering  is  in  order.  Well  then,  if  we  can’t  hope  to  hold  system 
configuration  stable  and  our  live  fire  test  &  evaluation  events  only  tell  us  about  a 
specific  configuration  and  a  specific  fire  solution  set,  what  are  we  to  do? 
Fortunately,  it  is  my  hypothesis  that  we  have  the  intellectual  ability  and  the 
technology  at  hand  to  address  this  issue  and  I  will  discuss  this  in  the  following 
paragraphs. 

V.  Proposed  Changes  to  the  Live  Fire  Test  and  Evaluation  Process. 

First  of  all,  abolition  of  the  live  fire  test  and  evaluation  process  is  not 
recommended.  Further,  a  reduction  of  live  fire  test  and  evaluation  funding  is 
not  necessarily  in  order,  although  new  processes  could  eventually  lead  to 
reductions  in  live  fire  test  and  evaluation  funding  requirements  at  some  point. 
This  paper’s  recommendations  will  take  the  form  of  business  process  re¬ 
engineering  and  management  changes.  There  are  no  apparent  reasons  why 
the  recommended  changes  can’t  take  place  within  the  context  of  the  existing 


29  From  an  interview  with  Lieutenant  Colonel  Jody  A.  Maxwell,  USA,  former  Product  Manager  of  the 
Apache  Longbow  Hellfire  Missile  and  recipient  of  the  1998  Daedalian  Award  for  his  performance  of  duty. 
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legislation,  policy  and  funding.  Therefore  there  is  no  apparent  requirement  to 
request  any  legislative  changes. 

First,  the  most  sensitive  changes  usually  involve  resource  reductions  to 
facilities,  manpower,  or  dollars.  Without  examining  the  long  term  impact  of  this 
paper’s  recommendations,  no  reductions  in  any  resources  dedicated  to  live 
fire  test  and  evaluation  process  are  recommended  at  this  time.  Indeed  the 
Director  of  Operational  Test  and  Evaluation  proposes  that  it  is  time  to  increase 
investments  in  Test  and  Evaluation  in  general.30  The  fundamental  issue  before 
us  is,  “What  is  the  best  way,  in  the  current  Performance  Based  Business 
Environment  and  Acquisition  Reform  environment,  to  ensure  combat  system 
survivability  and  munitions  lethality?”.  Restated,  “What  is  the  best  way  to  obtain 
a  high  degree  of  system  confidence  in  operational  suitability  and 
effectiveness?”  It  would  appear  that  funding  based  on  a  rate  of  0.3%  of 
procurement  resources  per  year  of  a  proposed  $60  billion  procurement  budget, 
or  about  $18  million,  is  not  exorbitant  and  is  furthermore  a  prudent  investment 
in  the  lives  of  our  service  people.  How  do  we  use  this  $18  million  to  obtain 
what  benefit  is  the  key  issue. 

Second,  live  fire  test  and  evaluation  program  responsibility  changes  are 
not  recommended.  The  independence  of  the  live  fire  test  and  evaluation 
community,  as  it  currently  stands,  is  a  prudent  management  structure.  It  is  a 

30  “Program  Manager  Interviews  Philip  Coyle”,  Program  Manager  Magazine,  May- June  1996,  pp.  2-8. 
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fundamentally  sound  approach  for  the  live  fire  test  and  evaluation  community  to 
remain  the  conscience  of  the  acquisition  system,  separate  from  the  materiel 
development  or  requirements  development  communities.  Even  though  the 
integrity  of  Defense  Program  Managers  is  correctly  presumed,  pressures  are 
strong  on  Program  Managers  to  get  their  systems  into  production  from  the 
service  users.  Often,  any  new  system  can  be  better  than  what  is  currently 
fielded.  Strong  pressure  can  be  brought  to  bear  on  Program  Managers  such 
that  there  exists  a  legitimate  viewpoint,  by  the  service  -  if  not  the  program 
manager,  that  it  is  more  desirable  to  field  a  flawed  system  and  fix  it  later,  than 
to  get  it  right  first.  This  inherent  conflict  of  interests  can’t  be  allowed  to  override 
the  well  being  of  our  volunteer  warriors. 

This  paper’s  first  recommendation  is  that  the  live  fire  test  and  evaluation 
community  must  effect  a  culture  shift  must  get  on  board  with  Acquisition 
Reform  now.  In  the  live  fire  community  there  appears  to  be  an  attitude  and 
culture  of,  “This  does  not  apply  to  us  and  we  are  exempt.”  Nothing  could  be 
further  from  the  truth.  It  applies  to  the  live  fire  community  whether  they  want  it  to 
or  not,  even  if  they  do  engage  in  wishful  thinking.  This  attitude  is  manifested  by 
comments  such  as  the  following: 

“We  have  no  evidence  to  show  us  that  you  can  substitute  modeling  and 
simulation  in  the  live  fire  world  for  live  fire  tests.”31 


31  John  Burt,  Defense  News,  August  1-7,  1994,  p.  22. 
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“It  is  a  waste  of  time  and  money  to  pursue  vulnerability  modeling  any  further. 
Experience  over  the  past  twenty  years  has  shown  that  we  have  been  unable  to 
validate  these  models.  Why  has  this  been  the  case?  The  events  to  occur 
inside  a  combat  configured  vehicle  that  has  been  struck  by  a  weapon  are 
complex,  non-linear  and  far  from  equilibrium.  These  are  the  characteristics  of 
chaotic  systems.  As  a  result  of  studies  in  this  new  non-linear  science,  we 
know  that  we  can  only  describe  complex,  non-linear  events  after  the  fact,  not 
before  the  fact.  More  computing  power  and  more  data  does  not  allow  us  to 
more  precisely  describe  what  has  already  occurred.  So  the  prospects  of  ever 
developing  and  validating  useful  vulnerability  models  seems  remote.”32 


“Simulation  Based  Acquisition  does  not  apply  to  Live  Fire  Test  &  Evaluation”33 


The  live  fire  test  and  evaluation  community  clearly  needs  to  get  with  the 
program.  To  think  that  the  rapidly  advancing  mathematical  fields  of  Chaos 
Theory  and  Non-Linear  Analysis  are  not  capable  of  ever  describing  impact 
dynamics  is  merely  wishful  thinking  at  best  and  arrogant  at  worst.  In  fact,  the 
same  arguments  the  live  fire  test  community  makes  against  simulation  can  be 
turned  against  them  given  the  highly  variable  nature  of  the  live  shot  business! 

Mathematical  modeling,  especially  when  used  in  concert  with  super- 
computing  and  parallel  computer  processing  capabilities  is  becoming  ever 
more  powerful.  Selected  models  may  be  inadequate  today,  but  they  should  be 
vigorously  pursued,  with  a  significant  portion  of  the  live  fire  test  and  evaluation 
resourcing  base  allocated  to  developing  the  mathematics,  the  models,  the 
interfaces  with  live  fire  data,  and  verification  and  validation.  This  is  an 


32  Colonel  James  Burton,  May  21,  1995  (and  subject  of  Home  Box  Office  Special  Motion  Picture, 

“ Pentagon  Wars”)  as  quoted  by  James  F.  O’Bryon,  slide  000299. 

33  James  F,  O’Bryon  as  quoted  by  Dr.  Jerry  Davis  at  the  1998  Simulation  Based  Acquisition  Conference. 
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especially  prudent  in  view  of  Moore’s  Law  which  states  that  computing  power 
doubles  about  every  eighteen  months. 

The  developmental  test  community  and  the  operational  test  community 
have  recognized  this  fact  of  life.  The  Army  has  done  significant  development  of 
the  “Virtual  Proving  Ground”  at  the  US  Army  Test  and  Evaluation  Command  with 
locations  around  the  world.  The  operational  test  community  is  putting 
significant  resources  into  system  on  system  and  force  on  force  models.  The 
Joint  Advanced  Distributed  Simulation  Joint  Test  Force,  located  in  Albuquerque, 
NM,  has  conducted  three  significant  tests  integrating  simulation  using  data 
baselined  from  actual  shot  data.  The  most  successful  and  promising  of  the 
three  tests  was  simulating  the  AMRAAM  missile  Precision  Guided  Munition. 

Simulation  is  not  cheap.  It  takes  resources  to  develop  models,  test  set  - 
ups,  and  interfaces  with  real  events.  The  test  simulation  architecture  for  the 
AMRAAM  test  cost  $600,000.  However,  each  AMRAAM  Missile  costs  $250,000 
not  counting  the  aircraft  and  test  infrastructure  support.34  This  simulation 
provided  the  capability  to  iterate  any  desired  number  of  AMRAAM  shots  within 
the  system  architecture,  once  established.  Further,  the  simulations  were 
based  on  live  shot  data.  While  it  is  true  that  there  were  some  constraints  and 
anomalies,  the  problems  seem  to  be  readily  solvable  with  a  modicum  of  effort. 


34  Lieutenant  Colonel  Pat  Cannon  and  Mr.  Eric  Keck,  “Advanced  Distributed  Simulation  -  A  Toll  for  the 
Tester’s  Toolbox”,  Joint  Advanced  Distributed  Simulation  Joint  Test  Force,  Albuquerque,  NM,  pp.  52-58, 
73-77,83-87,93,97-114. 
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Moreover,  it  does  not  take  too  many  live  AMRAAM  shots  (three  to  be  exact)  to 
exceed  the  cost  of  the  test  simulation'architecture,  which  provides  as  many 
shots  as  desired  and  affordable.  The  live  data  -simulation  linked  scenario  was 
certainly  more  effective  and  less  expensive  than  a  pure  live  fire  scenario  would 
have  been. 

There -is  hope  that  the  cultural  shift  can  occur.  A  leader  in  the  live  fire 
community,  Mr.  Jim  O’Bryon,  The  Deputy  Director  for  Live  Fire  Test  and 
Evaluation  at  the  Office  of  the  Secretary  of  Defense  states  in  his  briefings  that, 

“Testing  and  modeling  and  simulation  are  intertwined,  and  when  they  are  not, 
neither  is  effective.”35 

He  further  goes  on  to  offer  many  good  reasons  for  modeling  while 
mitigating  that  enthusiasm  by  offering  cogent  reasons  that  live  fire  tests  are 
essential.36 

Couple  the  mathematical  progress  with  increasing  computing  power 
and  it  will  not  be  long  until  simulations  can  describe  events  with  a  significant 
degree  of  accuracy.  Therefore,  the  second  recommendation  is  a  resource 
allocation  recommendation  to  dedicate  an  adequate  portion  of  the  live  fire  0.3% 
set  aside  of  the  procurement  budget  to  be  invested  in  live  fire  test  and 
evaluation  system  architecture  improvements,  model  developments  and 

35  James  F.  O’Bryon,  slide  000033. 

36  Ibid.,  slide  000300. 
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linkages  of  impact  dynamic  simulations  to  actual  test  event  data.  This  will  pay 
off  in  the  future  and  must  be  done  to  account  for  the  speed  with  internal 
configuration  of  systems  is  changing  under  acquisition  reform.  Otherwise  the 
secondary  effects  data  from  live  tests  events  (debris,  ricochet,  spalling,  etc.)  are 
only  valid  for  the  particular  configuration  tested  and  the  engagement  variable 
set  tested  and  therefore  of  limited  value  at  exorbitant  cost.  The  same  is  true  for 
synergistic  cascading  effects.  To  make  the  events  relevant  we  would  have  to 
build  two  of  every  configuration  change  and  destroy  one,  a  clearly  unaffordable 
approach,  and  would  still  be  valid  only  for  a  specific  set  of  engagement  criteria. 

The  third  recommendation  is  that  these  simulation  models  must  be 
linked  to  actual  live  fire  test  data.  This  live  fire  data  would  provide  a  baseline 
and  would  allow  simulation  excursions  to  account  for  changing  configurations. 
When  the  time  comes  for  another  live  fire  event,  then  the  simulation  can  be  re¬ 
baselined  or  validated.  Then  the  question  becomes,  how  will  we  know  when  it 
is  time  for  another  live  fire  event  to  obtain  new  baseline  data?  Restated,  when 
has  the  configuration  of  the  system  changed  so  significantly  that  it  is  rendering 
the  results  of  the  simulations  suspect  or  invalid?  The  answer  to  this  question 
is  the  key  objective  of  the  developed  simulation  models  and  iterations.  The 
effective  reform  of  the  live  fire  test  and  evaluation  process  would  thus  require 
both  a  reduced  number  of  live  shots  and  a  vastly  increased  number  of  effective 
simulation  iterations,  but  would  still  require  both  to  provide  value  and  relevancy. 
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Improving  the  models  and  linking  them  to  computing  power  is  “the  long 
pole  in  the  tent”  so  to  speak.  Linkage  requires  time,  research  and 
development  resourcing  and  should  be  started  immediately.  The  fourth 
recommendation  is  a  management  change  and  can  be  implemented  with  little 
or  no  resourcing.  Risk  management  techniques,  in  combination  with  the 
linked  simulation  and  actual  live  fire  event  data,  should  be  employed  to 
determine  the  milestones  for  live  fire  events.  We  know  that  as  system 
configuration  changes  from  that  which  was  tested  by  live  fire  event,  the 
relevance  of  a  simulation  would  decrease  over  time.  However,  we  need  not, 
and  indeed  cannot  afford  to,  test  every  configuration  change.  Risk  analysis 
must  be  done  to  determine  when  a  new  live  fire  event  is  required  when  the 
configuration  of  the  system  has  changed  so  significantly  that  the  results  of  the 
old  live  fire  event  -  simulation  linkage  are  largely  invalid.  Then  it  is  time  to 
schedule  and  resource  a  new  live  fire  event. 

The  principles  of  Risk  Management  are  well  known  in  the  Department  of 
Defense.  They  have  been  taught  for  years  at  the  Defense  Systems 
Management  College  and  have  taken  on  new  significance  in  the  environment 
of  Acquisition  Reform  and  resource  constraints.  One  of  the  foundations  is  Cost 
as  an  Independent  Variable.  This  concept  essentially  states  that  there  are 
risks  to  be  taken  and  risks  to  be  avoided  by  use  of  appropriate  risk 
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management  and  contingency  plans.37  The  establishment  of  a  Risk 
Management  Working  Group  that  included  representatives  of  the  Program 
Office,  Contractor,  User,  and  Live  Fire  Test  and  Evaluation  communities  is  also 
called  for  as  part  of  the  accepted  Integrated  Product  and  Process  Development 
concept 38  In  short ,  the  Live  Fire  Test  and  Evaluation  community  can  no  longer 
be  a  totally  detached  gate  keeper.  While  remaining  independent  of  the 
services  and  programs,  they  must  nevertheless  be  part  of  the  planning  and 
problem  solving  process.  This  Risk  Management  program  should  have  all  of 
the  following  components: 

•  Risk  Planning 

•  Risk  Assessment  (including  identification  and  analysis) 

•  Risk  Handling 

•  Risk  Monitoring 

Each  component  should  be  documented  so  that  assumption  of  risk  is 
known  and  agreed  to  by  the  proper  authority  level.39 


37  Mark  D.  Schaeffer,  “Risk  Management  in  the  Department  of  Defense:  Identifying  Risks  to  be  Taken  and 
Risks  to  be  Avoided”,  Program  Manager  Magazine,  March-April  1998,  p.48. 

38  Ibid.,  pp.  49-50. 

39  Ibid.,  p.52. 
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VI.  Summary. 


The  Revolution  in  Business  Affairs  is  now  six  to  eight  years  old.  The  only 
certainty  is  that  everything  in  defense  business  is  changing  and  at  an  ever 
accelerating  pace.  Acquisition  Reform  and  the  Performance  Based  Defense 
Business  environment  have  changed  almost  everything  about  the  way  we 
develop  and  procure  our  defense  weapons  systems.  The  advent  of  and  the 
emerging  maturity  of  the  simulation  based  acquisition  techniques  promise 
more  changes  in  the  future.  It  is  the  assessment  of  this  paper  that  only  one 
aspect  of  defense  acquisition  has  stood  firm  against  this  onslaught  of  change 
and  reform,  with  little  to  no  change  to  product  or  process,  and  that  would  be  the 
live  fire  test  and  evaluation  process  and  community.  This  can  not  continue  if 
this  key  community  is  to  retain  affordability  and  relevancy  to  the  defense 
acquisition  process,  which  it  must! 

The  tools  to  provide  a  better  live  fire  test  and  evaluation  process  and 
product  are  available  and  for  the  most  part  in  place.  Simulation  technology  is 
rapidly  maturing.  Computing  power  is  leaping  ahead  every  eighteen  months. 
Risk  management  techniques  are  well  developed  and  well  known  in  the 
Department  of  Defense.  The  live  fire  test  and  evaluation  community  has  their 
own  funding  and  they  have  authority  provided  to  them  by  law  and  by  policy.  Why 
then  is  there  no  evident  action  to  improve  the  live  fire  test  and  evaluation 
business  process  and  product  as  is  happening  in  every  other  aspect  defense 
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acquisition  and  test  and  evaluation,  like  the  advent  of  the  virtual  proving 
ground?40  There  does  not  appear  to  be  an  answer  to  this  question  short  of 
saying  it  is  nice  to  be  in  a  comfort  zone.  The  live  fire  test  and  evaluation 
community  must  establish  a  vision  and  a  plan,  including  resourcing,  for 
relevancy  in  the  future,  or  they  will  become  irrelevant.  My  hope  is  that  this 
changes  in  the  very  near  term.  Perhaps  this  paper  can  be  a  catalyst.  The  live 
fire  test  and  evaluation  program  is  too  important  to  the  well  being  of  our 
warriors.  As  important  as  the  process  is,  it  can  not  continue  spending  millions 
for  dubious  “value  added”  output.  It  is  my  sincere  hope  that  the  change  has 
already  begun  and  that  I  simply  could  not  find  it  and  that  the  community  leaders 
are  re-engineering  this  critical  aspect  of  defense  acquisition  as  this  paper  is 
being  written. 


40  Virtual  Proving  Ground,  http://vp9.ap2.armv.mil 
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